Add DeriveKeyPair API #1877

Eddy-M-K · 2024-07-31T14:22:12Z

Add derandomized keypair generation and encapsulation API for all KEM schemes (only ML-KEM has an internal implementation at the moment, all other schemes will throw an error).

Closes #1206

Does this PR change the input/output behaviour of a cryptographic algorithm (i.e., does it change known answer test values)? (If so, a version bump will be required from x.y.z to x.(y+1).0.)
Does this PR change the list of algorithms available -- either adding, removing, or renaming? Does this PR otherwise change an API? (If so, PRs in fully supported downstream projects dependent on these, i.e., oqs-provider will also need to be ready for review and merge by the time this is merged.)

keypair_derand() and encaps_derand() are added to the API of all KEM schemes.

Signed-off-by: Eddy Kim <[email protected]> Signed-off-by: Eddy Kim <[email protected]>

Signed-off-by: Eddy Kim <[email protected]>

Signed-off-by: Eddy Kim <[email protected]> Signed-off-by: Eddy Kim <[email protected]>

- Add support for BIKE, FrodoKEM, sntrup - Add hooks for testing - Add missing kem comment to documentation - Don't run decaps() in test_kem_derand if encaps_derand() fails - Add markdown documentation changes Signed-off-by: Eddy Kim <[email protected]>

Signed-off-by: Eddy Kim <[email protected]>

baentsch · 2024-07-31T17:16:05Z

First off: Thanks for the work making this available, @Eddy-M-K -- and for more than just one algorithm!

If I understand it right, you're adding two new functions to the KEM algs, thus requiring all consumers of the library that want to make use of this to write lots of new code (essentially replacing/augmenting keygen/encaps with their new de-randomizing twins): Did I get this right? Is there no way to avoid that (say, by creating a setup function changing the semantics of keygen/encaps)? Is it really necessary to set "randomness"/well, "seeds" at every API invocation?

baentsch · 2024-08-01T06:51:06Z

src/kem/kem.h

+     * @param[in] coins The input randomness represented as a byte string.
+	 * @return OQS_SUCCESS or OQS_ERROR
+	 */
+	OQS_STATUS (*encaps_derand)(uint8_t *ciphertext, uint8_t *shared_secret, const uint8_t *public_key, const uint8_t *coins);


OK -- now I understand some more: The underlying implementation asks for the seed/coins to be passed in each call. This however begs the question: How many "coins" are needed to correctly operate this API? Shouldn't this be documented & checked here? Equivalent question: How many coins are needed for the keygen API call? Beyond the ability to improve API robustness (check this at invocation), if this were known(?) wouldn't it be possible to pass a sufficient number of "coins" to a "setup_random_coins" call that is invoked once and then used "under the covers" for each subsequent call to (API-wise unchanged) keygen and encaps calls? That way the many "OQS_ERROR" return responses for the KEM algs not supporting that could be saved: They simply wouldn't use "prepared coins" when asked to operate keygen&encaps.

Sorry it's taken me so long to follow up on this, Michael.

For the first question, length_keypair_coins and length_encaps_coins are added to the OQS_KEM struct; the caller can use them similarly to length_shared_secret, length_ciphertext, etc.

For the second, first let me confirm I'm understanding correctly: setup_random_coins would be a one-time call to set up (i.e., seed) a source of randomness which would then be used as RNG for OQS_randombytes in subsequent KEM operations? This would be similar in functionality to the OQS_randombytes_custom_algorithm call but wouldn't require the caller to define their own function and pass it as a pointer—correct?

The purpose of this PR is to achieve something different. The keypair_derand call is intended to implement (well, expose) the DeriveKeyPair KEM function defined in RFC 9180. This is used notably in the recent Messaging Layer Security standard. By design, this is basically a KDF for KEM keys—the caller provides input keying material and gets back a KEM keypair—which needs to be called multiple times rather than having a one-time setup. The intent is to supplement the existing keygen function, not replace it: RFC 9180 requires that the KEM supports both operations. ML-KEM already supports the DeriveKeyPair operation as part of its API via keypair_derand, as well as a similar encaps_derand function. (I don't know an application for the latter off the top of my head.) I imagine it would have been nice to have this background before you looked at the PR; sorry for the poor coordination on my part.

Thanks for the explanations. I now understand some more -- but clearly not everything: I thought this PR is meant to enable open-quantum-safe/oqs-provider#447? If this were so (?), I'm afraid this API will require substantial changes to oqsprovider to facilitate this, no? But then again, maybe you or @Eddy-M-K can show me wrong by doing a simple PR implementing open-quantum-safe/oqs-provider#447 based on this new API (maybe a "tracking-..." branch to trigger CI as we also do with new algs?). Again, will be online only by smart phone until Aug 19, so please do not expect serious further feedback from me until then.

bifurcation · 2024-11-21T15:07:05Z

@Eddy-M-K @baentsch - What do we need to do to get this PR un-stuck? Support for HPKE is also blocked on the lack of DeriveKeyPair.

FWIW: I would simplify this PR some:

Remove the encaps_derand() parts, since they're not required for DeriveKeyPair
Don't change documentation for non-upgraded KEMs; if DeriveKeyPair isn't supported, there's no need to talk about its parameters

Also, a naming nit: Instead of _keypair_derand, I would use _derive_keypair. AFAIK, the derand terminology is not widely used; _derive_keypair would at least align terminology with HPKE, which is the primary consumer and deployed on some scale.

SWilson4 · 2024-11-22T16:37:11Z

@Eddy-M-K @baentsch - What do we need to do to get this PR un-stuck? Support for HPKE is also blocked on the lack of DeriveKeyPair.

Truthfully, this has stalled mostly to our limited developer time and other features being higher priority (it's not a coincidence that the last activity on this PR was the Friday before the NIST final standards were released). If you are interested in getting this PR over the finish line, the help would be more than welcome.

One open question is the oqs-provider integration: I had hoped that we could kill two birds with one stone and use the new API to solve open-quantum-safe/oqs-provider#447. For HPKE, would you be looking to use liboqs APIs directly or go via the OpenSSL3 provider?

FWIW: I would simplify this PR some:

* Remove the `encaps_derand()` parts, since they're not required for DeriveKeyPair

* Don't change documentation for non-upgraded KEMs; if DeriveKeyPair isn't supported, there's no need to talk about its parameters

I would have no objection to removing encaps_derand. I'm not sure about the latter point, as we have yet to implement an API function that is only supported by one algorithm and generally aim for uniform support (see the spirited discussion on #1919). I don't believe that we would be able to achieve the latter goal for this feature, however.

Also, a naming nit: Instead of _keypair_derand, I would use _derive_keypair. AFAIK, the derand terminology is not widely used; _derive_keypair would at least align terminology with HPKE, which is the primary consumer and deployed on some scale.

I have no objection to this, either.

I do have one question as I revisit this PR after the publication of FIPS 203: can we expose the "derand" (which I suppose is now "internal") API without running afoul of the final standard? This paragraph at the top of page 16 makes me wary:

Controlled access to internal functions. The key-encapsulation mechanism ML-KEM makes use
of internal, “derandomized” functions ML-KEM.KeyGen_internal, ML-KEM.Encaps_internal, and
ML-KEM.Decaps_internal, specified in Section 6. The interfaces for these functions should not
be made available to applications other than for testing purposes. In particular, the sampling of
random values required for key generation (as specified in ML-KEM.KeyGen) and encapsulation
(as specified in ML-KEM.Encaps) shall be performed by the cryptographic module.

baentsch · 2024-11-22T17:09:17Z

I do have one question as I revisit this PR after the publication of FIPS 203: can we expose the "derand" (which I suppose is now "internal") API without running afoul of the final standard?

Welcome to the club @SWilson4 : See e.g., openssl/openssl#25938 (comment) now put into documentation in openssl/openssl#26037. Looks like there really is an advantage contributing to different projects aiming at solving the same problem... :-)

Eddy-M-K and others added 29 commits July 31, 2024 10:24

Initial derive keypair commit

9ee89ac

Signed-off-by: Eddy Kim <[email protected]> Signed-off-by: Eddy Kim <[email protected]>

Add pqcrystals-ml_kem_ipd.patch

854e036

Signed-off-by: Eddy Kim <[email protected]> Signed-off-by: Eddy Kim <[email protected]>

Fix encaps key in scheme and revert whitespace changes

f11c479

Signed-off-by: Eddy Kim <[email protected]>

Hopefully corrected patch file

0b41866

Signed-off-by: Eddy Kim <[email protected]>

Corrected missing derand in kem_scheme

de45506

Signed-off-by: Eddy Kim <[email protected]>

Fix indentation

9e6bb19

Signed-off-by: Eddy Kim <[email protected]>

Run copy_from_upstream

7ce72d7

Signed-off-by: Eddy Kim <[email protected]>

derand testing tentative changes

60bfc5e

Signed-off-by: Eddy Kim <[email protected]>

Add missing function declarations

008161f

Signed-off-by: Eddy Kim <[email protected]>

Add template for avx2 derand functions

85685ff

Signed-off-by: Eddy Kim <[email protected]>

Run copy_from_upstream

f985af6

Signed-off-by: Eddy Kim <[email protected]>

WIP: Add changes for coin length

2b83af3

Signed-off-by: Eddy Kim <[email protected]>

Update patch to include coin lengths

bc37415

Signed-off-by: Eddy Kim <[email protected]>

Bootstrap

56c0087

Signed-off-by: Eddy Kim <[email protected]>

fixup! Update patch to include coin lengths

981e9eb

Signed-off-by: Eddy Kim <[email protected]>

Conditional copy

c7f0608

Signed-off-by: Eddy Kim <[email protected]>

Run copy_from_upstream

d9eb0ed

Signed-off-by: Eddy Kim <[email protected]>

Separate coins variable into two distinct variables

7370f17

Signed-off-by: Eddy Kim <[email protected]> Signed-off-by: Eddy Kim <[email protected]>

Add derand fixes

9609bbc

- Add support for BIKE, FrodoKEM, sntrup - Add hooks for testing - Add missing kem comment to documentation - Don't run decaps() in test_kem_derand if encaps_derand() fails - Add markdown documentation changes Signed-off-by: Eddy Kim <[email protected]>

WIP trying to fix build errors

04bdea0

Signed-off-by: Eddy Kim <[email protected]>

Fix remaining build issues

9b639b7

Signed-off-by: Eddy Kim <[email protected]>

Resolve unused parameter issues for BIKE

6f296d8

Signed-off-by: Eddy Kim <[email protected]>

Resolve unused paramter issues for FrodoKEM

8c06c20

Signed-off-by: Eddy Kim <[email protected]>

Fix whitespace inconsistency

6eec29a

Signed-off-by: Eddy Kim <[email protected]>

Fix whitepace issue

6537493

Signed-off-by: Eddy Kim <[email protected]>

Insert unused attributes

2406568

Signed-off-by: Eddy Kim <[email protected]>

Void all unused parameters

68435ad

Signed-off-by: Eddy Kim <[email protected]>

Use tab instead of spaces in kem_scheme

6790657

Signed-off-by: Eddy Kim <[email protected]>

Run copy_from_upstream

f4fb753

Signed-off-by: Eddy Kim <[email protected]>

Eddy-M-K force-pushed the ek-derive-keypair-api branch from ec52093 to f4fb753 Compare July 31, 2024 14:24

SWilson4 mentioned this pull request Jul 31, 2024

Support deterministic key generation open-quantum-safe/oqs-provider#447

Open

Eddy-M-K added 2 commits July 31, 2024 10:41

Fix kem_derand python tests

bf435c2

Signed-off-by: Eddy Kim <[email protected]>

Initialize coins in test_kem_derand

4cbb475

Signed-off-by: Eddy Kim <[email protected]>

baentsch reviewed Aug 1, 2024

View reviewed changes

dstebila added this to the 0.12.0 milestone Aug 14, 2024

dstebila mentioned this pull request Nov 12, 2024

Support 64-byte seed form of ML-KEM private keys #1985

Closed

dstebila modified the milestones: 0.12.0, 0.13.0 Nov 19, 2024

SWilson4 mentioned this pull request Nov 20, 2024

Support derivation of KEM key pairs #2003

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add DeriveKeyPair API #1877

Add DeriveKeyPair API #1877

Eddy-M-K commented Jul 31, 2024

baentsch commented Jul 31, 2024

baentsch Aug 1, 2024

SWilson4 Aug 9, 2024 •

edited

Loading

baentsch Aug 9, 2024

bifurcation commented Nov 21, 2024

SWilson4 commented Nov 22, 2024

baentsch commented Nov 22, 2024

Add DeriveKeyPair API #1877

Are you sure you want to change the base?

Add DeriveKeyPair API #1877

Conversation

Eddy-M-K commented Jul 31, 2024

baentsch commented Jul 31, 2024

baentsch Aug 1, 2024

Choose a reason for hiding this comment

SWilson4 Aug 9, 2024 • edited Loading

Choose a reason for hiding this comment

baentsch Aug 9, 2024

Choose a reason for hiding this comment

bifurcation commented Nov 21, 2024

SWilson4 commented Nov 22, 2024

baentsch commented Nov 22, 2024

SWilson4 Aug 9, 2024 •

edited

Loading